Chinese Word Segmentation based on Maximum Matching and Word Binding Force

l)okfulam ih)a,d thmg Kong pkwong((~cs.hku.hk and Abstract A Chinese word Seglnentation algorithm based on forward icnaxinnlln matching and word binding force is t)roposed in this pai)er. This algorithm Iilays a key role in post-processing the outtmt of a character or st/eech recognizer in determining the proper word sequence c(/rre-st)onding to an input line of cha.raeter images or a speech wav(~,fol'tn. ~FO support this algorithm, a text; (:orims of over 63 millions characters is employed to enrich an 80,O00-words lexi(:on in terlns of its word entries and word binding forces. As it stands now, given an input line of text, the word segmentor can proce, ss on the average 210,000 characters per se(:-ond when running on an IBM RISC Sys-tem/6000 3BT workstation with a col rect word identitication rate of 99.74%. 1 Introduction A language model as a t)ost-processor is esse, ntial to a recognizer of speech or characters in order to determine the approi)riate word se, que, n(:e and henc.e the semantics of an inI)ut line of text or utterance. It is well known that an N-gram statistics language model is just as effective as, t)ut nmch more eificient than, a syntactk:/semantic analyser in determining the correct word sequence. A necessary condition to successflfl collection of N-gram statistics is the existence of a coInprehensive le, x-icon and a large text corpus. The latter must tie lexically analysed in order to identify all the words, from which, N-gram statistics can be derived. About 5,000 characters are being used in modern Chinese and they are the building blocks of all wor(ls. Ahnost every character is a word and inost words are of one or two characters long but there are also abundant wor(ls longer than two characters. Before it; is seginented into words, a line of text is just a sequence of characters and there are numerous word segmentation alternatives. Usually , all but one of these alternatives arc syntactically and/or semantically incorrect. This is l;he case because unlike texts in English, Chinese texl;s have no word nlarkers. A tirst step towmds buiht-ing a language model based on N-gram statistics is to de, vek)p an etIMent lexical analyser to id(!ntify all the words in the, corpus. Word segmentation algorithlns behmg to one of two types ill general, viz., the structural (Wang et al. rt;spec-tively. A structural algorithm resolves segmenta-tion mnbiguities by examining the structural rcla-tionships between words, while a statistical algorithm compares …